## [1] "Excluded 1 participants based on catch-trial performance."
We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.
## `summarise()` has grouped output by 'modal'. You can override using the
## `.groups` argument.
## `summarise()` has grouped output by 'modal', 'percentage_blue'. You can
## override using the `.groups` argument.
## [1] "Excluded 0 participants based on random responses."
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.
We use the AUC function with the splines
method to directly compute the AUC.
t-test and regression model with control variables:
##
## Two Sample t-test
##
## data: aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = 2.9517, df = 122, p-value = 0.003791
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 3.969716 20.138118
## sample estimates:
## mean of x mean of y
## 20.20750 8.15358
## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula:
## auc_diff ~ cond + test_order + first_speaker_type + confident_speaker +
## first_speaker_type * cond + (1 | workerid)
## Data: auc_d
##
## REML criterion at convergence: 1086.2
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.1828 -0.4581 0.0770 0.6374 2.0920
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 89.25 9.447
## Residual 376.89 19.414
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error df t value Pr(>|t|)
## (Intercept) 14.152 2.118 58.000 6.684 9.95e-09 ***
## cond1 6.027 1.743 60.000 3.457 0.00101 **
## test_order1 0.878 2.122 58.000 0.414 0.68056
## first_speaker_type1 -7.666 2.118 58.000 -3.620 0.00062 ***
## confident_speaker1 2.544 2.122 58.000 1.199 0.23544
## cond1:first_speaker_type1 1.211 1.743 60.000 0.695 0.48990
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cond1 tst_r1 frs__1 cnfd_1
## cond1 0.000
## test_order1 -0.032 0.000
## frst_spkr_1 0.000 0.000 0.002
## cnfdnt_spk1 0.002 0.000 -0.065 -0.032
## cnd1:frs__1 0.000 0.000 0.000 0.000 0.000
library(mclust)
## Package 'mclust' version 5.4.10
## Type 'citation("mclust")' for citing this R package in publications.
##
## Attaching package: 'mclust'
## The following object is masked from 'package:DescTools':
##
## BrierScore
## The following object is masked from 'package:bootstrap':
##
## diabetes
aucs_diff = merge(aucs.cautious, aucs.confident, by=c("workerid"))
aucs_diff$diff_of_diffs = aucs_diff$auc_diff.x - aucs_diff$auc_diff.y
aucs_diff %>% ggplot(aes(x=diff_of_diffs)) + geom_density() + geom_jitter(aes(y=0), width=0, height=0.001) + ggtitle("Raw data + estimated density")
1 Cluster
fit1 = Mclust(aucs_diff$diff_of_diffs, G=1)
print(summary(fit1, parameters=2))
## ----------------------------------------------------
## Gaussian finite mixture model fitted by EM algorithm
## ----------------------------------------------------
##
## Mclust X (univariate normal) model with 1 component:
##
## log-likelihood n df BIC ICL
## -292.5844 62 2 -593.423 -593.423
##
## Clustering table:
## 1
## 62
##
## Mixing probabilities:
## 1
## 1
##
## Means:
## [1] 12.05392
##
## Variances:
## [1] 735.3371
2 Clusters
fit2 = Mclust(aucs_diff$diff_of_diffs, G=2)
print(summary(fit2, parameters=T))
## ----------------------------------------------------
## Gaussian finite mixture model fitted by EM algorithm
## ----------------------------------------------------
##
## Mclust E (univariate, equal variance) model with 2 components:
##
## log-likelihood n df BIC ICL
## -288.1198 62 4 -592.7482 -598.9994
##
## Clustering table:
## 1 2
## 55 7
##
## Mixing probabilities:
## 1 2
## 0.864934 0.135066
##
## Means:
## 1 2
## 4.335758 61.479346
##
## Variances:
## 1 2
## 353.8638 353.8638
3 Clusters
fit3 = Mclust(aucs_diff$diff_of_diffs, G=3)
print(summary(fit3, parameters=T))
## ----------------------------------------------------
## Gaussian finite mixture model fitted by EM algorithm
## ----------------------------------------------------
##
## Mclust E (univariate, equal variance) model with 3 components:
##
## log-likelihood n df BIC ICL
## -288.1317 62 6 -601.0263 -656.7775
##
## Clustering table:
## 1 2 3
## 9 46 7
##
## Mixing probabilities:
## 1 2 3
## 0.3373576 0.5314408 0.1312016
##
## Means:
## 1 2 3
## -1.081510 7.998816 62.254385
##
## Variances:
## 1 2 3
## 337.7512 337.7512 337.7512
According to the Bayesian information criterion, a model with two clusters describes the data best.
Fitted model:
aucs_diff %>%
ggplot(aes(x=diff_of_diffs)) +
geom_jitter(aes(y=0, color=first_speaker_type.x), width=0, height=0.001) +
ggtitle("Raw data + Components of gaussian mixture") +
stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[1], sd = sqrt(fit2$parameters$variance$sigmasq[1]))) +
stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[2], sd = sqrt(fit2$parameters$variance$sigmasq[2])))
## Warning: Removed 101 row(s) containing missing values (geom_path).
## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: most_likely_model ~ condition + test_order + first_speaker_type +
## first_speaker_type * condition + (1 | workerid)
## Data: d.post_test
##
## AIC BIC logLik deviance df.resid
## 154.1 171.0 -71.0 142.1 118
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.6887 -0.5686 -0.2379 0.5566 2.2970
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 1.849 1.36
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error z value
## (Intercept) -0.4481 0.3106 -1.443
## conditioncautious -0.8698 0.2906 -2.993
## test_orderparallel -0.4682 0.3081 -1.520
## first_speaker_typecautious 0.8680 0.3414 2.542
## conditioncautious:first_speaker_typecautious -0.2066 0.2423 -0.852
## Pr(>|z|)
## (Intercept) 0.14906
## conditioncautious 0.00276 **
## test_orderparallel 0.12856
## first_speaker_typecautious 0.01102 *
## conditioncautious:first_speaker_typecautious 0.39396
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cndtnc tst_rd frst__
## conditincts 0.209
## tst_rdrprll 0.106 0.220
## frst_spkr_t -0.193 -0.363 -0.198
## cndtncts:__ -0.039 0.080 0.055 -0.062
## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +
## first_speaker_type * condition + (1 | workerid)
## Data: d.post_test
##
## REML criterion at convergence: 1596.8
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.2024 -0.6789 -0.0483 0.5532 2.6459
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 2007 44.8
## Residual 30231 173.9
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error df t value
## (Intercept) -14.48 16.63 59.00 -0.871
## conditioncautious -56.74 15.61 60.00 -3.634
## test_orderparallel -25.21 16.63 59.00 -1.516
## first_speaker_typecautious 56.68 16.62 59.00 3.410
## conditioncautious:first_speaker_typecautious -13.92 15.61 60.00 -0.891
## Pr(>|t|)
## (Intercept) 0.38744
## conditioncautious 0.00058 ***
## test_orderparallel 0.13477
## first_speaker_typecautious 0.00118 **
## conditioncautious:first_speaker_typecautious 0.37634
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cndtnc tst_rd frst__
## conditincts 0.000
## tst_rdrprll -0.032 0.000
## frst_spkr_t 0.000 0.000 0.000
## cndtncts:__ 0.000 0.000 0.000 0.000
## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +
## prior_likelihood_ratio + first_speaker_type * condition +
## (1 | workerid)
## Data: d.post_test
##
## REML criterion at convergence: 1587.1
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.39818 -0.66926 -0.04121 0.51524 3.06625
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 0 0.0
## Residual 29267 171.1
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error df
## (Intercept) 16.4539 17.6008 118.0000
## conditioncautious -56.7419 15.3630 118.0000
## test_orderparallel -20.5737 15.4247 118.0000
## first_speaker_typecautious 49.0672 15.5072 118.0000
## prior_likelihood_ratio 0.4202 0.1165 118.0000
## conditioncautious:first_speaker_typecautious -13.9165 15.3630 118.0000
## t value Pr(>|t|)
## (Intercept) 0.935 0.351779
## conditioncautious -3.693 0.000337 ***
## test_orderparallel -1.334 0.184835
## first_speaker_typecautious 3.164 0.001979 **
## prior_likelihood_ratio 3.607 0.000455 ***
## conditioncautious:first_speaker_typecautious -0.906 0.366866
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cndtnc tst_rd frst__ prr_l_
## conditincts 0.000
## tst_rdrprll 0.013 0.000
## frst_spkr_t -0.066 0.000 -0.011
## prr_lklhd_r 0.487 0.000 0.083 -0.136
## cndtncts:__ 0.000 0.000 0.000 0.000 0.000
## optimizer (nloptwrap) convergence code: 0 (OK)
## boundary (singular) fit: see help('isSingular')
## Data: d.post_test
## Models:
## model1: likelihood_ratio ~ condition + test_order + first_speaker_type + first_speaker_type * condition + (1 | workerid)
## model2: likelihood_ratio ~ condition + test_order + first_speaker_type + prior_likelihood_ratio + first_speaker_type * condition + (1 | workerid)
## npar AIC BIC logLik deviance Chisq Df Pr(>Chisq)
## model1 7 1647.8 1667.5 -816.89 1633.8
## model2 8 1637.0 1659.5 -810.49 1621.0 12.792 1 0.0003481 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
| workerid | first_speaker_type | test_order | noticed_manipulation | cautious_count | confident_count | aligned_count | first_adaptation_speaker_count |
|---|---|---|---|---|---|---|---|
| 1352 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1353 | confident | parallel | 0 | 1 | 1 | 2 | 1 |
| 1354 | cautious | parallel | 0 | 1 | 1 | 2 | 1 |
| 1355 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| 1359 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| 1365 | cautious | reverse | 0 | 1 | 1 | 2 | 1 |
| 1368 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1370 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1373 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1375 | confident | reverse | 1 | 1 | 1 | 2 | 1 |
| 1385 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1387 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1391 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1395 | cautious | parallel | 1 | 1 | 1 | 2 | 1 |
| 1397 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1407 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| 1417 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1423 | cautious | parallel | 1 | 1 | 1 | 2 | 1 |
| 1428 | cautious | parallel | 1 | 1 | 1 | 2 | 1 |
| 1432 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| workerid | first_speaker_type | test_order | noticed_manipulation | cautious_count | confident_count | aligned_count | first_adaptation_speaker_count |
|---|---|---|---|---|---|---|---|
| 1404 | confident | parallel | 1 | 1 | 1 | 0 | 1 |
| 1406 | cautious | parallel | 0 | 1 | 1 | 0 | 1 |
| 1419 | confident | reverse | 0 | 1 | 1 | 0 | 1 |